HPS: a hierarchical Persian stemming method

In this paper, a novel hierarchical Persian Stemming approach based on the Part-Of-Speech (POS) of the word in a sentence is presented. The implemented stemmer includes hash tables and several deterministic finite automata (DFA) in its different levels of hierarchy for removing the prefixes and suffixes of the words. We had two intentions in using hash tables in our method. The first one is that the DFA don’t support some special words, so hash table can partly solve the addressed problem. And the second goal is to speed up the implemented stemmer with omitting the time that DFA need. Because of the hierarchical organization, this method is fast and flexible enough. Our experiments on test sets from Hamshahri Collection and Security News from ICTna.ir Site show that our method has the average accuracy of 95.37% which is even improved in using the method on a test set with common topics.